Experiments with Online Reinforcement Learning in Real-Time Strategy Games

Real-Time Strategy (RTS) games provide a challenging platform to implement online reinforcement learning (RL) techniques in a real application. Computer as one player monitors opponents’ (human or other computers) strategies and then updates its own policy using RL methods. In this paper, we firstly examine the suitability of applying the online RL in various computer games. RL application depends much on both RL complexity and the game features. We then propose a multi-layer framework for implementing online RL in an RTS game. The framework significantly reduces RL computational complexity by decomposing the state space in a hierarchical manner. We implement an RTS game Tank General, and perform a thorough test on the proposed framework. We consider three typical profiles of RTS game players and compare two basic RL techniques applied in the game. The results show the effectiveness of our proposed framework and shed light on relevant issues on using online RL in RTS games.